{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "bc0db63f-8d18-4929-9535-edf58ae15e62",
   "metadata": {},
   "source": [
    "## LLaMA 2 指令微调（Alpaca-Style on Dolly-15K Dataset)\n",
    "\n",
    "示例代码关键训练要素：\n",
    "- 使用 Dolly-15K 数据集，以 Alpaca 指令风格生成训练数据\n",
    "- 以 4-bit（NF4）量化精度加载 `LLaMA 2-7B` 模型\n",
    "- 使用 QLoRA 以 `bf16` 混合精度训练模型\n",
    "- 使用 `HuggingFace TRL` 的 `SFTTrainer` 实现监督指令微调\n",
    "- 使用 Flash Attention 快速注意力机制加速训练（需硬件支持）"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e97de889-4d1d-4ee3-ad57-289e2e5919d8",
   "metadata": {},
   "source": [
    "### 下载 databricks-dolly-15k 数据集"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "ef235f0d-df0d-4be9-a034-af6684588dbe",
   "metadata": {},
   "outputs": [],
   "source": [
    "from datasets import load_dataset\n",
    "from random import randrange\n",
    " \n",
    "# 从hub加载数据集\n",
    "dataset = load_dataset(\"databricks/databricks-dolly-15k\", split=\"train\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "4c5f398a-6dd4-4f71-a474-1b647103471a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Dataset({\n",
       "    features: ['instruction', 'context', 'response', 'category'],\n",
       "    num_rows: 15011\n",
       "})"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 数据集样例总数: 15011\n",
    "dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "61788bc1-6c19-4301-9089-14a59c23b869",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'instruction': 'you are riding your bicycle to the store and your bicycle has a basket. which of the following items can you take back home? a toothbrush, a TV, a bar of soap, a pair of pants, a pair of skis, a loaf of bread, a tiger, a phone charger', 'context': '', 'response': 'you can take back, the tooth brush, the bar of soap, a pair of pants, a loaf of bread and a phone charger', 'category': 'classification'}\n"
     ]
    }
   ],
   "source": [
    "# 随机抽选一个数据样例打印\n",
    "print(dataset[randrange(len(dataset))])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "71040ca1-92d0-4e1e-af66-49ef2a52c3d7",
   "metadata": {},
   "source": [
    "### 以 Alpaca-Style 格式化指令数据\n",
    "\n",
    "`Alpacca-style` 格式：https://github.com/tatsu-lab/stanford_alpaca#data-release"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "9d2889f7-ff56-4ee5-a852-1794c0a296ac",
   "metadata": {},
   "outputs": [],
   "source": [
    "def format_instruction(sample_data):\n",
    "    \"\"\"\n",
    "    Formats the given data into a structured instruction format.\n",
    "\n",
    "    Parameters:\n",
    "    sample_data (dict): A dictionary containing 'response' and 'instruction' keys.\n",
    "\n",
    "    Returns:\n",
    "    str: A formatted string containing the instruction, input, and response.\n",
    "    \"\"\"\n",
    "    # Check if required keys exist in the sample_data\n",
    "    if 'response' not in sample_data or 'instruction' not in sample_data:\n",
    "        # Handle the error or return a default message\n",
    "        return \"Error: 'response' or 'instruction' key missing in the input data.\"\n",
    "\n",
    "    return f\"\"\"### Instruction:\n",
    "Use the Input below to create an instruction, which could have been used to generate the input using an LLM. \n",
    " \n",
    "### Input:\n",
    "{sample_data['response']}\n",
    " \n",
    "### Response:\n",
    "{sample_data['instruction']}\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "26167b60-f339-487a-a2dc-0cf49fac8932",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "### Instruction:\n",
      "Use the Input below to create an instruction, which could have been used to generate the input using an LLM. \n",
      " \n",
      "### Input:\n",
      "The largest airlines in the world can be defined in several ways. As of 2022, Delta Air Lines is the largest by revenue, assets value and market capitalization, China Southern Air Holding by passengers carried, American Airlines Group by revenue passenger mile, fleet size, numbers of employees and destinations served, FedEx Express by freight tonne-kilometers, Ryanair by number of routes, Turkish Airlines by number of countries served.\n",
      " \n",
      "### Response:\n",
      "What is the largest airline in the world ?\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# 随机抽选一个样例，打印 Alpaca 格式化后的样例 \n",
    "print(format_instruction(dataset[randrange(len(dataset))]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "109b77d2-58e2-4018-a943-51bbfd68ee4e",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "2eea57c8-ffac-422c-9fa1-b4b98dd3f917",
   "metadata": {},
   "source": [
    "### 使用快速注意力（Flash Attention）加速训练\n",
    "\n",
    "检查你的 GPU 是否支持 `flash-attn` 加速：\n",
    "\n",
    "```shell\n",
    "$ python -c \"import torch; assert torch.cuda.get_device_capability()[0] >= 8, 'Hardware not supported for Flash Attention'\"\n",
    "\n",
    "Traceback (most recent call last):\n",
    "  File \"<string>\", line 1, in <module>\n",
    "AssertionError: Hardware not supported for Flash Attention\n",
    "```\n",
    "**运行结果：演示使用的 NVIDIA T4 硬件不支持 Flash Attention**\n",
    "\n",
    "#### 安装 flash-attn 加速包（需要GPU硬件支持）\n",
    "\n",
    "```shell\n",
    "$ MAX_JOBS=4 pip install flash-attn --no-build-isolation\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "45a715dc-3ecd-41bb-9b2e-88c1db991e92",
   "metadata": {},
   "source": [
    "### 加载模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "677fa03d-439e-4aa1-81f0-e1015d71c189",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7438362070ef4d84a2e4981e717c12f1",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/root/miniconda3/lib/python3.11/site-packages/transformers/generation/configuration_utils.py:392: UserWarning: `do_sample` is set to `False`. However, `temperature` is set to `0.9` -- this flag is only used in sample-based generation modes. You should set `do_sample=True` or unset `temperature`. This was detected when initializing the generation config instance, which means the corresponding file may hold incorrect parameterization and should be fixed.\n",
      "  warnings.warn(\n",
      "/root/miniconda3/lib/python3.11/site-packages/transformers/generation/configuration_utils.py:397: UserWarning: `do_sample` is set to `False`. However, `top_p` is set to `0.6` -- this flag is only used in sample-based generation modes. You should set `do_sample=True` or unset `top_p`. This was detected when initializing the generation config instance, which means the corresponding file may hold incorrect parameterization and should be fixed.\n",
      "  warnings.warn(\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig\n",
    "\n",
    "# 如果硬件设备支持，成功安装 flash-attn后，将 use_flash_attention 设置为True\n",
    "use_flash_attention = False\n",
    " \n",
    "# 取消注释以使用 flash-atten\n",
    "# if torch.cuda.get_device_capability()[0] >= 8:\n",
    "#     from utils.llama_patch import replace_attn_with_flash_attn\n",
    "#     print(\"Using flash attention\")\n",
    "#     replace_attn_with_flash_attn()\n",
    "#     use_flash_attention = True\n",
    " \n",
    " \n",
    "# 获取 LLaMA 2-7B 模型权重\n",
    "# 无需 Meta AI 审核的模型权重\n",
    "model_id = \"NousResearch/Llama-2-7b-hf\" \n",
    "# 通过 Meta AI 审核后可使用此 Model ID 下载\n",
    "# model_id = \"meta-llama/Llama-2-7b-hf\" \n",
    " \n",
    " \n",
    "# 使用 BnB 加载量化后的模型\n",
    "bnb_config = BitsAndBytesConfig(\n",
    "    load_in_4bit=True,\n",
    "    bnb_4bit_use_double_quant=True,\n",
    "    bnb_4bit_quant_type=\"nf4\",\n",
    "    bnb_4bit_compute_dtype=torch.bfloat16\n",
    ")\n",
    " \n",
    "# 加载模型与分词器\n",
    "model = AutoModelForCausalLM.from_pretrained(model_id, quantization_config=bnb_config, use_cache=False, device_map=\"auto\")\n",
    "model.config.pretraining_tp = 1 \n",
    " \n",
    "# 通过对比doc中的字符串，验证模型是否在使用flash attention\n",
    "if use_flash_attention:\n",
    "    from utils.llama_patch import forward    \n",
    "    assert model.model.layers[0].self_attn.forward.__doc__ == forward.__doc__, \"Model is not using flash attention\"\n",
    " \n",
    " \n",
    "tokenizer = AutoTokenizer.from_pretrained(model_id)\n",
    "tokenizer.pad_token = tokenizer.eos_token\n",
    "tokenizer.padding_side = \"right\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "782b1b42-97f2-4378-99d3-d582fae48e03",
   "metadata": {},
   "source": [
    "### 使用 QLoRA 配置加载 PEFT 模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "50690552-1807-4e53-892a-1aad10cadaca",
   "metadata": {},
   "outputs": [],
   "source": [
    "from peft import LoraConfig, prepare_model_for_kbit_training, get_peft_model\n",
    " \n",
    "# QLoRA 配置\n",
    "peft_config = LoraConfig(\n",
    "        lora_alpha=16,\n",
    "        lora_dropout=0.1,\n",
    "        r=16,\n",
    "        bias=\"none\",\n",
    "        task_type=\"CAUSAL_LM\", \n",
    ")\n",
    " \n",
    " \n",
    "# 使用 QLoRA 配置加载 PEFT 模型\n",
    "model = prepare_model_for_kbit_training(model)\n",
    "qlora_model = get_peft_model(model, peft_config)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "98490c6a-fa9d-460b-8362-33444efd2e8a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "trainable params: 8,388,608 || all params: 6,746,804,224 || trainable%: 0.12433454005023165\n"
     ]
    }
   ],
   "source": [
    "qlora_model.print_trainable_parameters()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f2c0f7b6-30a3-4e10-8a4a-9840e465e590",
   "metadata": {},
   "source": [
    "### 训练超参数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "eb427d32-575f-4af7-af71-bbcfc0b3b730",
   "metadata": {},
   "outputs": [],
   "source": [
    "import datetime\n",
    "\n",
    "timestamp = datetime.datetime.now().strftime(\"%Y%m%d_%H%M%S\")\n",
    "\n",
    "# 演示训练参数（实际训练是设置为 False）\n",
    "demo_train = True\n",
    "output_dir = f\"models/llama-7-int4-dolly-{timestamp}\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "1f57027d-63cc-4199-9ae7-d04fa9db3f2b",
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import TrainingArguments\n",
    " \n",
    "args = TrainingArguments(\n",
    "    output_dir=output_dir,\n",
    "    num_train_epochs=1 if demo_train else 3,\n",
    "    max_steps=100,\n",
    "    per_device_train_batch_size=3, # Nvidia T4 16GB 显存支持的最大 Batch Size\n",
    "    gradient_accumulation_steps=1 if demo_train else 4,\n",
    "    gradient_checkpointing=True,\n",
    "    optim=\"paged_adamw_32bit\",\n",
    "    logging_steps=10,\n",
    "    save_strategy=\"steps\" if demo_train else \"epoch\",\n",
    "    save_steps=10,\n",
    "    learning_rate=2e-4,\n",
    "    bf16=True,\n",
    "    max_grad_norm=0.3,\n",
    "    warmup_ratio=0.03,\n",
    "    lr_scheduler_type=\"constant\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "49e28e91-6daa-44c9-8e1e-59abddd57697",
   "metadata": {},
   "source": [
    "### 实例化 SFTTrainer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "53c29ebe-f962-4b2f-9905-743f63024cf0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[2024-03-09 00:05:43,454] [INFO] [real_accelerator.py:191:get_accelerator] Setting ds_accelerator to cuda (auto detect)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Detected kernel version 4.4.0, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher.\n",
      "/root/miniconda3/lib/python3.11/site-packages/trl/trainer/sft_trainer.py:310: UserWarning: You passed `packing=True` to the SFTTrainer, and you are training your model with `max_steps` strategy. The dataset will be iterated until the `max_steps` are reached.\n",
      "  warnings.warn(\n"
     ]
    }
   ],
   "source": [
    "from trl import SFTTrainer\n",
    " \n",
    "# 数据集的最大长度序列（筛选后的训练数据样例数为1158）\n",
    "max_seq_length = 2048 \n",
    " \n",
    "trainer = SFTTrainer(\n",
    "    model=qlora_model,\n",
    "    train_dataset=dataset,\n",
    "    peft_config=peft_config,\n",
    "    max_seq_length=max_seq_length,\n",
    "    tokenizer=tokenizer,\n",
    "    packing=True,\n",
    "    formatting_func=format_instruction, \n",
    "    args=args,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c716498f-93c7-4071-858e-c53d270eeb2d",
   "metadata": {},
   "source": [
    "### 训练模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "43d07ef2-d07c-41c1-8eaa-2472f289055c",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/root/miniconda3/lib/python3.11/site-packages/torch/utils/checkpoint.py:464: UserWarning: torch.utils.checkpoint: the use_reentrant parameter should be passed explicitly. In version 2.4 we will raise an exception if use_reentrant is not passed. use_reentrant=False is recommended, but if you need to preserve the current default behavior, you can pass use_reentrant=True. Refer to docs for more details on the differences between the two variants.\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "      \n",
       "      <progress value='100' max='100' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [100/100 3:09:33, Epoch 0/1]\n",
       "    </div>\n",
       "    <table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       " <tr style=\"text-align: left;\">\n",
       "      <th>Step</th>\n",
       "      <th>Training Loss</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>1.561400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>20</td>\n",
       "      <td>1.421200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>30</td>\n",
       "      <td>1.338200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>40</td>\n",
       "      <td>1.276900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>50</td>\n",
       "      <td>1.252400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>60</td>\n",
       "      <td>1.221700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>70</td>\n",
       "      <td>1.274100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>80</td>\n",
       "      <td>1.223100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>90</td>\n",
       "      <td>1.244000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>100</td>\n",
       "      <td>1.180300</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table><p>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Checkpoint destination directory models/llama-7-int4-dolly/checkpoint-10 already exists and is non-empty.Saving will proceed but saved results may be invalid.\n",
      "/root/miniconda3/lib/python3.11/site-packages/torch/utils/checkpoint.py:464: UserWarning: torch.utils.checkpoint: the use_reentrant parameter should be passed explicitly. In version 2.4 we will raise an exception if use_reentrant is not passed. use_reentrant=False is recommended, but if you need to preserve the current default behavior, you can pass use_reentrant=True. Refer to docs for more details on the differences between the two variants.\n",
      "  warnings.warn(\n",
      "Checkpoint destination directory models/llama-7-int4-dolly/checkpoint-20 already exists and is non-empty.Saving will proceed but saved results may be invalid.\n",
      "/root/miniconda3/lib/python3.11/site-packages/torch/utils/checkpoint.py:464: UserWarning: torch.utils.checkpoint: the use_reentrant parameter should be passed explicitly. In version 2.4 we will raise an exception if use_reentrant is not passed. use_reentrant=False is recommended, but if you need to preserve the current default behavior, you can pass use_reentrant=True. Refer to docs for more details on the differences between the two variants.\n",
      "  warnings.warn(\n",
      "/root/miniconda3/lib/python3.11/site-packages/torch/utils/checkpoint.py:464: UserWarning: torch.utils.checkpoint: the use_reentrant parameter should be passed explicitly. In version 2.4 we will raise an exception if use_reentrant is not passed. use_reentrant=False is recommended, but if you need to preserve the current default behavior, you can pass use_reentrant=True. Refer to docs for more details on the differences between the two variants.\n",
      "  warnings.warn(\n",
      "/root/miniconda3/lib/python3.11/site-packages/torch/utils/checkpoint.py:464: UserWarning: torch.utils.checkpoint: the use_reentrant parameter should be passed explicitly. In version 2.4 we will raise an exception if use_reentrant is not passed. use_reentrant=False is recommended, but if you need to preserve the current default behavior, you can pass use_reentrant=True. Refer to docs for more details on the differences between the two variants.\n",
      "  warnings.warn(\n",
      "/root/miniconda3/lib/python3.11/site-packages/torch/utils/checkpoint.py:464: UserWarning: torch.utils.checkpoint: the use_reentrant parameter should be passed explicitly. In version 2.4 we will raise an exception if use_reentrant is not passed. use_reentrant=False is recommended, but if you need to preserve the current default behavior, you can pass use_reentrant=True. Refer to docs for more details on the differences between the two variants.\n",
      "  warnings.warn(\n",
      "/root/miniconda3/lib/python3.11/site-packages/torch/utils/checkpoint.py:464: UserWarning: torch.utils.checkpoint: the use_reentrant parameter should be passed explicitly. In version 2.4 we will raise an exception if use_reentrant is not passed. use_reentrant=False is recommended, but if you need to preserve the current default behavior, you can pass use_reentrant=True. Refer to docs for more details on the differences between the two variants.\n",
      "  warnings.warn(\n",
      "/root/miniconda3/lib/python3.11/site-packages/torch/utils/checkpoint.py:464: UserWarning: torch.utils.checkpoint: the use_reentrant parameter should be passed explicitly. In version 2.4 we will raise an exception if use_reentrant is not passed. use_reentrant=False is recommended, but if you need to preserve the current default behavior, you can pass use_reentrant=True. Refer to docs for more details on the differences between the two variants.\n",
      "  warnings.warn(\n",
      "/root/miniconda3/lib/python3.11/site-packages/torch/utils/checkpoint.py:464: UserWarning: torch.utils.checkpoint: the use_reentrant parameter should be passed explicitly. In version 2.4 we will raise an exception if use_reentrant is not passed. use_reentrant=False is recommended, but if you need to preserve the current default behavior, you can pass use_reentrant=True. Refer to docs for more details on the differences between the two variants.\n",
      "  warnings.warn(\n",
      "/root/miniconda3/lib/python3.11/site-packages/torch/utils/checkpoint.py:464: UserWarning: torch.utils.checkpoint: the use_reentrant parameter should be passed explicitly. In version 2.4 we will raise an exception if use_reentrant is not passed. use_reentrant=False is recommended, but if you need to preserve the current default behavior, you can pass use_reentrant=True. Refer to docs for more details on the differences between the two variants.\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "TrainOutput(global_step=100, training_loss=1.299329252243042, metrics={'train_runtime': 11487.4223, 'train_samples_per_second': 0.026, 'train_steps_per_second': 0.009, 'total_flos': 2.43882352705536e+16, 'train_loss': 1.299329252243042, 'epoch': 0.26})"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trainer.train()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fd5eeb5e-2082-4ead-b4a2-ebedb06333f9",
   "metadata": {},
   "source": [
    "### 保存模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "e874176b-1b1a-4264-b24e-5308c773e81d",
   "metadata": {},
   "outputs": [],
   "source": [
    "trainer.save_model()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d39256f7-6b10-4f6d-8a99-c96a512604ec",
   "metadata": {},
   "source": [
    "### 模型推理（测试）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5e38d38d-a675-40a1-aa0d-b78e90865162",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
